:- module(WebDB,
        [main/0], [assertions, regtypes, basicmodes]). 

:- use_module(library('compiler/c_itf')).
:- use_module(library('assertions/assrt_lib')).

% :- use_module(library(basicprops)).

%% ---------------------------------------------------------------------------
%% Intro
%% ---------------------------------------------------------------------------

:- comment(title,"The WebDB WWW Database Interface").

:- comment(subtitle,"A Customizable Database Interface through Word Wide Web").
:- comment(subtitle,"An Application of Persistent Predicates, Pillow and Active Modules"). % Delete?
:- comment(subtitle,"@bf{The CIAO System Documentation Series}").
:- comment(subtitle,"Technical Report CLIP 11/98.0").
:- comment(subtitle,"RadioWeb (ESPRIT Project 25562) Report D3.1.M1-A3").
:- comment(subtitle,"ECCOSIC (@em{Comision Conjunta Hispano-Norteamericana}").
:- comment(subtitle,"@em{de Cooperacion Cientifica y Tecnologica} Project 98059)"). 
:- comment(subtitle,"@em{Draft printed on:} @today{}").

:- comment(author, "J.M. Gomez, D. Cabeza and M. Hermenegildo").
:- comment(author, "@tt{clip@@dia.fi.upm.es}").
:- comment(author, "@tt{http://www.clip.dia.fi.upm.es/}").
:- comment(author, "The CLIP Group").
:- comment(author, "Facultad de Inform@'{a}tica").
:- comment(author, "Universidad Polit@'{e}cnica de Madrid").

:- comment(copyright,"
Copyright @copyright{} 1997-98 The Clip Group.

@include{Copyright.Manuals}
").

:- comment(summary, "@include{README.lpdoc}

This document also constitutes an @em{internals} manual, providing
information on how the different internal parts of @apl{WebDB} are
connected.").

:- comment(module,"

@apl{WebDB} provides a way to create and customize a @index{Relational
Database} and/or a @index{WWW interface} to access it. 
@apl{WebDB} provides a simple WWW interface based on @index{HTML} forms 
which allows the user to access the data stored in any database, local to
@apl{WebDB} itself or remote.

In the first case, the tables are designed according to the definition
of filebased persistent predicates, using the library @lib{persdb}
@cite{radioweb-D3.1.M2-A1} and are maintained directly by @apl{WebDB},
allowing the system administrator to create, destroy or edit the tables.
In the second case, the data is stored in remote SQL databases, so that
@apl{WebDB} just provides an interface to them through the primitives
espeficied in the library @lib{persdb_sql} @cite{radioweb-D3.1.M2-A2}.

As we will see, the information can be accessed in two different ways. We
could say that @apl{WebDB} provides a @index{single table interface} and
a @index{multitable interface}. 

Through the first way, the interface to access a table is created automatically
on every access. This is achieved by defining each table field according to an
@index{html_format} type, which is translated by a template into the
right HTML input type for the given field, for example, a checkbox, an
input box or a textarea. 

The system administrator can add new types and define tables according to
them. Of course, @apl{WebDB} provides an interface to develope this task in
an automatic way. 

The @concept{multitable interface} or, as we will see in further sections,
the complex query option provides an interface to make queries over a
database view in both, Prolog or SQL syntax, by offering a window where the
user can write them.

Settings in @apl{WebDB} are kept in @index{configuration files}, and
are customizable, as well as types and templates. The system
administrator can customize them on line, without needing to stop the
system, through the interface provided by the application itself.

@section{Data Storage}

As mentioned above, the data @apl{WebDB} operates on can be local or
located at remote SQL databases. In both cases, @apl{WebDB} provides a
WWW interface to access it in a transparent and comfortable way.

These two kinds of data storage are based upon the same concept:
@concept{Persistent Predicates}. The idea is that whatever changes might
have been done on one of these predicates survive across executions. This
way, no matter whether the system stops, or even crashes, we will always have a
consistent state of the predicate, which is quite an useful thing if we
want to implement a database. If Prolog is halted and restarted, the
predicate the new Prolog process sees is right in the state it was in
the very moment the old process was halted, provided that no
changes to the external storage were done in the meantime by other
processes or the user himself.

Actually, there are two different implementations of @concept{persistent
predicates}:
@begin{itemize}
@item File based persistent predicates, which local tables are based upon, and

@item SQL based persistent predicates, which is the way @apl{WebDB}
 implements the tables located on remote SQL databases.
@end{itemize}

The first method can be used for storing smaller amounts of data that we
will call @index{local data}. In @apl{WebDB} this kind of persistent
predicates are also used to manage most of the internal information the
system needs to handle the operations the services it provides resemble. In
a wider context, e.g. the Prolog world, it seems an easy and efficient way
to achieve persistence, saving the state of the predicates through
executions, which, up to now, is something Prolog systems where
quite lack of.

The second method provides a very high-level natural way to access, 
from a Prolog program, larger amounts of data in SQL database
relations. According to this method, persistent predicates are linked to
tables in external databases so that the predicates actually reside in
such tables. These tables are seen from the Prolog side as ordinary
predicates.

In the next figure, we can appreciate the interaction between
@apl{WebDB} and these two kinds of persistent predicates as well as the
system's architecture:

@image{architecture}

As we can see in the figure, the @apl{WebDB} server provides its
services through the WWW. These services basically consist of an
interface to query the database and, in the case of the system
administrator, manage the system.

@section{The Generic Database Server: dbserver}
This part of the @apl{WebDB} documentation is aimed to describe its database 
server, @apl{dbserver}, which provides the users with 
services to access a given database. This server has been developed as a 
@apl{CIAO prolog} @index{active module} and is basically an executable which 
publishes its IP address in order to allow client programs to use its services.
Client programs are invoked with the CGI protocol by the web server and
connect to the active module to request server actions.

This server also uses the capabilities provided by the filebased
persistent predicates to manage the multiple aspects of the database it
is bound to, for example, to keep the current state of a search or the
current description and type of a given table, or even the text messages
which are output as a result of any interaction with the system.

@subsection{Launching dbserver}

In order to make the database accessible from the web, the dbserver is
launched through the CGI @file{startdbserver.cgi} and its standard error
output is redirected to a file called @file{dbserver.log}, located on
the @file{private} directory. This way, any misbehaviour or problem
occurred will be registered and, therefore, solved more easily. The
steps the dbserver carries out at this point follow:

First of all, the dbserver processes the contents of the file
@file{dbserver_ini.pl}.  This file is created during the installation and
contains important data relative to the directories of the database and
its URLs.

The next step is to read the configuration files for the database.
Database customization basically lies on these files, which
contain information regarding the @concept{html_format} @index{types}
and @index{templates} and their translation into @lib{PiLLoW} terms,
as well as information about the searches lifetime, the translation
between @concept{html_format} and SQL @concept{types} and the remote SQL
databases the system is bound to. 

Once all this information has been processed, dbserver can start the
processing of data regarding the database tables. So, dbserver
enables for future queries both kinds of tables, local tables based upon
filebased persistent predicates and remote tables based upon SQL-based
persistent predicates.

The next step is to start the statistics procedure by resetting the date of
initialization for the dbserver. This information can be accessed through
the service provided by the option @pred{status} lying on the maintenance front
 page .

Finally, if necessary, dbserver generates a @tt{maintenance front page}
which is the first part of the manager interface, and initializes all those
other persistent predicates embedded in the system which are used for
inner tasks such as keeping a trace of the current state of queries and
the text of the user interface messages.

@section{WebDB Interface}

The @apl{WebDB} interface is aimed to be an easy and straightforward way to
query and manage a database. As we can see in the figure, this interface
is composed by several stages. Each one offers the right interface for
a given service, many of them only allowed to the user administrator.

@image{Interface}

@subsection{Manager Interface}

Let's concentrate on the manager interface as the common user interface
is quite simple and consists of a restricted version of the manager
search interface.

The first stage of the manager or administrator interface is a
maintenance front page where the following options are provided:

@begin{itemize}
@item @em{Start the System}, obviously used to start the dbserver.
@item @em{System Status}, which provides information about the dbserver
status and statistics about the system's usage.
@item @em{Remote SQL Databases Connection Management} is the door to an
interface which enables the system administrator to connect
@apl{WebDB} to remote SQL databases and unlink it from a previously added
one. Through this interface, it is possible to collect the information
@apl{WebDB} needs in order to accessing the tables of a given
database. Besides, it provides information about the remote SQL databases
the system is already linked to.
@item @em{Inspect/Insert/Modify Data}, provides an interface where the user 
 can operate on each table of the database individually.
@item @em{Complex Query}, provides another interface to operate on the 
 database. This interface is quite similar to a Prolog top level, an 
 environment where the user can make his own queries according to Prolog
and SQL syntax standards. 
@item @em{Modify Messages and Text Settings}, provides a list with all
 the system messages and the text settings and allows the system
 administrator to edit them.
@item @em{Create/Destroy Tables}, allows the system administrator to
 create new filebased tables for the current database or destroy any of
 the already existing as well as unlink the system from remote SQL tables. 
@item @em{Edit Tables Types}, allows the system administrator to edit
 the type of any field of a filebased table or to add or delete them.
@item @em{Reload Templates and Update DB Types}. Once the
 @concept{configuration files} have been changed, this option reloads
 them and updates the system with the new configuration.
@item @em{Stop the System}, halts the system and updates the information
stored upon filebased persistent predicates.
@end{itemize}

Additional information must be given about the fourth and fifth options: 
@subsection{The Inspect/Insert/Modify Data option}

Once the user has decided on which table to operate on, this option
displays the system consult interface. Through this interface the user
can choose which operation to carry out.

Initially, the system administrator, can choose among the following operations:
@begin{itemize}
@item @em{Search}: Sequential search. The results of the query are
 displayed one by one as the user requests them.
@item @em{Overview}: All the tuples that instantiate the query are
 displayed in a table, but just as prolog facts, without the additional
 information supplied by the fields types.
@item @em{Full search}: As in the case of the @em{Overview} operation,
 all the tuples that instantiate the query are displayed, but in this
 case with the additional information of its field types. Besides,
 through this option, the user can edit the results too.
@item @em{Insert element}: Allows the user to insert a new element,
 which has to be completely specified, e.g. no field of the form can
 remain empty. Besides, the system tests that the element to be inserted
 doesn't exist in the table already. 
@item @em{Delete element}: Deletes all the tuples that instantiate the query.
@item @em{Modify element}: Allows the user to edit a given tuple of the table.
@end{itemize}
If the user is not an authorized client, the operations allowed will only be 
@em{Search}, @em{Overview} and @em{Full search} which, in this case,
will not allow the user to edit the results of a previous query. 

Note that when any query is made over a database table, the information
written into the form fields is taken as prolog terms. This way, any
empty field of the form is translated into a free variable and therefore
will unify all the corresponding arguments of the given table.

Besides, @apl{WebDB} provides three different search modes:
@begin{itemize}
@item @em{Normal search mode}:  This is the most restrictive search
mode. The results of the query will be just those tuples which
exactly match the fields specified.
@item @em{Case insensitive search mode}: In this mode upper and lower
case is indistinct.
@item @em{Wildcards search mode}: Implements a search where
wildcards can be used. Special characters are:
 @begin{description}
  @item{*} Matches any string, including the null string.
  @item{?} Matches any single character.
  @item{[...]} Matches any one of the enclosed characters.  A pair of
   characters separated by a minus sign denotes a range; any character
   lexically between those two characters, inclusive, is matched.  If the
   first character following the [ is a ^ then any character not enclosed
   is matched.  No other character is special inside this construct.  To
   include a ] in a character set, you must make it the first character.
   To include a `-', you must use it in a context where it cannot possibly
   indicate a range: that is, as the first character, or immediately after
   a range.
  @item{|} specifies an alternative.  Two regular expressions A and B with
   | in between form an expression that matches anything that either A or B
   will match.
  @item{@{...@}} groups alternatives inside larger patterns.
  @item{\} Quotes a special character (including itself).
 @end{description}
@end{itemize}

@subsection{The Complex Query option}
This option is fully available for both authorised and not authorised
clients. As mentioned above, it provides an interface to query the database
in Prolog and SQL syntax. It is necessary to select the view the tables
involved in the query lay on.  

When only local tables are involved in the query, the predicates allowed in
this environment are restricted in order to avoid any eventual
misuse. This set of allowed predicates can be classified in the next
categories: arithmetic and relational operators, connectors (i.e. ',' and
';') and the tables of the database itself.  

On the other hand, in the case of tables belonging to remote SQL databases,
it is quite recommendable to control which views are to be accessible for the 
system users and with which rights in order to  prevent the SQL tables
from being manipulated unduly. 

As an on-line guide for the user, this option displays a list of the tables
the database consists of. This includes information about the arity of the
tables, the physical database where they are located and the types of their
fields. 

There are three possible answers for a complex query:
@begin{itemize}
@item @em{No elements found} if there were no matches for the query.
@item @em{Unsuccesful query} if the query tried to attempt against the system
integrity or any predicate involved didn't exist.
@item @em{Complex Query Response}. In this case the output is a table with the
name of every variable written in the query and their values.
@end{itemize}

@subsection{Clients of the Generic WebDB Distribution}

This part of the @apl{WebDB} documentation is aimed to describe the CGI
clients which are included by default with the distribution of
@apl{WebDB} and form the interface described in the previous section.

These clients use the services provided by the server to query the
database or manage the system. From this point on, the number of clients
added only depends on the system administrator and the number and nature
of the services required by any specific case.

Each client is firmly related to the services provided by the server. In
most cases, this relation is an injective application, as we can see:

@begin{itemize}
@item @em{dbcommonselection}  <->  @pred{get_relationsdb} 
@item @em{dbcommonclient}     <->  @pred{consultdb}  
@item @em{dbselection}        <->  @pred{get_relationsdb} 
@item @em{dbclient}           <->  @pred{consultdb}  
@item @em{dbcomplexquery}     <->  @pred{complex_query}  
@item @em{dbcreatetable}      <->  @pred{create_destroy_table}  
@item @em{dbeditselection}    <->  @pred{select_table_to_edit}  
@item @em{dbeditclient}       <->  @pred{edit_table}  
@item @em{dboptions}          <->  @pred{consultdboptions}  
@item @em{dbtemplates}        <->  @pred{dbtemplates}  
@item @em{dbserverstatus}     <->  @pred{status}  
@item @em{dbconnect_DB}	      <->  @pred{connect_to_DB}  
@item @em{dbserverstop}       <->  @pred{dying}, @pred{die}  
@end{itemize}

Going back to the first figure, we can see that, in fact, every
component of @apl{WebDB} can be located on distant hosts on the WWW, as
they get through with one another using the @concept{HTTP} protocol.
This leaves an open door for distributed databases due to its
interconnectivity.

@section{WebDB Types and Templates}

@apl{WebDB}'s customizability lies on a series of files, file based
persistent predicates and predicates  which constitute a system of types and 
templates in order to generate the HTML representation of every table. In the 
nex figure we can appreciate the interaction of each of these components
with one another and with dbserver.


@image{templates}

Let's describe these components:
@begin{itemize}

@item @em{html_options} are a set of .html files processed by the dbserver
 both when it is started and when the system administrator reloads the
 templates of the system through the operation @em{Reload Templates and
 Update DB Types}, available at the maintenance front page. These
 templates are parametric. So, the user can set them to the right values
 and customize the database.

 It is important to bear in mind that although these files are
 parametric, their structure, as well as their name, have to be
 respected.  Moreover, the user must not delete any of them.

 So, in order to customize the settings, from the background color to the
 representation of the html tables, the user only needs to change the
 parameters of these files and reload them, using the previously
 mentioned operation.

@item @em{text_options} is one of the persistent predicates embedded in
 the system mentioned in previous sections. It is used to store all the
 messages and text the system uses when interacting with a client. As we
 can see in the figure, this component is linked to dbserver in both
 ways. This means that, although dbserver uses it to produce the right
 text and messages, it is also able to edit them. To do so, the system
 administrator must choose the @em{Modify Messages and Text Settings}
 operation.

@item @em{dbtables_desc} is the next of these system embedded persistent
 predicates. It stores the definition of all the tables the database
 consists of, both remote and local.  To define a table, the system
 administrator has two options:
@begin{itemize}
@item Create it through the interface provided by @apl{WebDB} using the
 operations @em{Create/Destroy Tables} and @em{Edit Tables Types} of the
 maintenance front page.

@item Create it by adding an entry in the file 
@file{DBDATADESC/db_desc/dbserver:dbtables_desc-2.pl}. This entry must
be written in the following format:
@begin{verbatim}
'dbserver:dbtables_desc'( 
<<Table_name>>(
	<<Type 1>>(<<Field description 1>>, <<Length 1>>), 
	<<Type 2>>(<<Field description 2>>, <<Length 2>>), 
	
	<<Type n>>(<<Field description n>>, <<Length n>>)), 
	<<Situation>>,
	<<Creation Date>>).
@end{verbatim}

Besides, it is possible to link the tables laying in an SQL remote database 
through the operation @em{Remote SQL Databases Connection Management}.

@apl{WebDB} distribution includes an example of these types and
templates for a given database and surely the best way to make a new one
is by pattern matching.

Each of these types need to have their corresponding entry in the file 
@file{html_format} located on the directory @file{DBDATADESC/db_types} which 
will be described next.

In every case, the second argument of @pred{dbserver:dbtables_desc/3}
points out whether the table is local, by setting that field to the
value `local', or remote, by writing the identifier of the view for the 
SQL database. That kind of identifier is defined in the filebased persistent 
predicate @pred{sql_location/2} which, just like the 
@file{DBDATADESC/db_types/html2sql.pl}, is described in the next section
(Defining SQL remote Databases).

@end{itemize}

@item @pred{type_info/3} and @pred{type_val_info/4} are used to obtain the
 translation between a type defined for a field in @em{dbtables_desc}
 and its corresponding template of @em{html_format}, which is the
 component to be described next.  @pred{type_info} is used to get the
 @em{html_format} template of a table field and its description before
 making any query and, on the other hand, @pred{type_val_info} gets the
 contents of a table field after the query has been made.

@item{html_format} is a @index{macro} which provides an interface to
 use the primitives provided by the library @lib{PiLLoW} in a
 transparent way. An entry of @em{html_format} can be equivalent to a
 large piece of @lib{PiLLoW} code and, therefore, it contributes to make
 things simpler and more customizable.
@end{itemize}

@section{Defining SQL remote Databases}
This is fundamentally achieved thanks to the library @lib{persdb_sql} which 
provides and interface to remote SQL databases located on Win95/NT
machines. In order to use the facilities provided by that library, @apl{WebDB} needs the predicate @pred{sql_location/2} and the file @file{html2sql.pl}, 
described next:

@begin{itemize}

@item @pred{sql_location.pl} is an SQL based persistent predicate that 
contains entries @apl{WebDB} uses to define the fact 
@pred{sql_persistent_location/2} (see the documentation of @lib{persdb_sql} 
@cite{radioweb-D3.1.M2-A2}) for each remote SQL database view linked to 
@apl{WebDB}.

@item @file{html2sql.pl} contains instances of the predicate html2sql/2 which 
provide the translation between the macros defined in @em{html_format} and 
their equivalent SQL types.
@item @file{sql2html.pl}, in the opposite direction, allows to find the right 
html_dormat representation type for a given SQL type.
@end{itemize}

").

main.

% -----------------------------------------------------------------------------

:- comment(version_maintenance,dir('../version')).

:- comment(version(0*5+12,1999/11/06,22:02*26+'MET'), "Updated
   documentation to lpdoc-1.9 formats.  (Manuel Hermenegildo)").

:- comment(version(0*5+11,1999/09/16,12:52*05+'MEST'), "Added lock file to
   prevent the system from being started several times (Jose Manuel Gomez
   Perez)").

:- comment(version(0*5+10,1999/09/03,10:58*16+'MEST'), "Generic client
   interface added (Jose Manuel Gomez Perez)").

:- comment(version(0*5+9,1999/06/01,19:35*14+'MEST'), "Automatic
   distribution procedure completed.  (Manuel Hermenegildo)").

:- comment(version(0*5+8,1999/05/31,17:11*11+'MEST'), "Readmes now automatically generated from documentation.  (Manuel Hermenegildo)").  

:- comment(version(0*5+7,1999/04/23,13:14*44+'MEST'), "Naive graphical
statistics added (Jose Manuel Gomez Perez)").

:- comment(version(0*5+6,1999/04/21,11:58*39+'MEST'), "Text statistics
added (Jose Manuel Gomez Perez)").

:- comment(version(0*5+5,1999/04/20,13:09*27+'MEST'), "Infrastructure for
system status info added (Jose Manuel Gomez Perez)").

:- comment(version(0*5+4,1999/03/29,19:57*47+'MEST'), "HTML templates
revised (Jose Manuel Gomez Perez)").

:- comment(version(0*5+3,1999/03/23,18:58*21+'MET'), "Automatic link to
remote SQL databases through ODBC added (Jose Manuel Gomez Perez)").

:- comment(version(0*5+2,1999/02/10,12:29*33+'MET'), "Corrected lack of
generality at the edit tables interface (Jose Manuel Gomez Perez)").

:- comment(version(0*5+1,1999/02/10,12:28*51+'MET'), "Complex query with
restricted preds. completed (Jose Manuel Gomez Perez)").

:- comment(version(0*5+0,1998/07/23,13:26*00+'MET DST'), "Started
   autodocumentation.  (Manuel Hermenegildo)").

:- comment(version(0*1+1,1998/3/18), "Updated maintenance interface.  (Jose
   Manuel Gomez Perez)").

:- comment(version(0*1+0,1998/3/18), "Added recursive treatment of
   relations (base case).  (Jose Manuel Gomez Perez)").

:- comment(version(0*1+0,1998/2/20), "Started automatic
   documentation. (Jose Manuel Gomez Perez)").

% -----------------------------------------------------------------------------


